enable `django-structlog`'s celery integration by Alc-Alc · Pull Request #3199 · python/pythondotorg

Alc-Alc · 2026-10-25T11:26:09Z

Description

Implemented the suggestion in structure the logs #3188 (comment)
does what is mentioned in https://django-structlog.readthedocs.io/en/latest/celery.html#getting-started-with-celery

Closes

nothing (but will make @JacobCoffee and @jrobichaud happy)

Before / After

Success - before

2026-10-25T11:05:17.689092Z [info     ] Task pydotorg.celery.run_management_command[6ed93a24-fed0-4f24-b9ee-56a018f938cd] received [celery.worker.strategy] data={'id': '6ed93a24-fed0-4f24-b9ee-56a018f938cd', 'name': 'pydotorg.celery.run_management_command', 'args': "['check', [], {}]", 'kwargs': '{}', 'eta': None}
System check identified no issues (0 silenced).
2026-10-25T11:05:17.730706Z [info     ] Task pydotorg.celery.run_management_command[6ed93a24-fed0-4f24-b9ee-56a018f938cd] succeeded in 0.03981362499689567s: None [celery.app.trace] data={'id': '6ed93a24-fed0-4f24-b9ee-56a018f938cd', 'name': 'pydotorg.celery.run_management_command', 'return_value': 'None', 'runtime': 0.03981362499689567, 'args': "['check', [], {}]", 'kwargs': '{}'}

Success - after

2026-10-25T11:07:23.501008Z [info     ] Task pydotorg.celery.run_management_command[8d79611d-38c9-4be2-8a0f-2f3cf5b950f2] received [celery.worker.strategy] data={'id': '8d79611d-38c9-4be2-8a0f-2f3cf5b950f2', 'name': 'pydotorg.celery.run_management_command', 'args': "['check', [], {}]", 'kwargs': '{}', 'eta': None}
2026-10-25T11:07:23.502808Z [info     ] task_started                   [django_structlog.celery.receivers] request_id=e01e8ff3-9a27-44e9-887e-02da37d3a291 task=pydotorg.celery.run_management_command task_id=8d79611d-38c9-4be2-8a0f-2f3cf5b950f2 user_id=3
System check identified no issues (0 silenced).
2026-10-25T11:07:23.537175Z [info     ] task_succeeded                 [django_structlog.celery.receivers] duration_ms=35 request_id=e01e8ff3-9a27-44e9-887e-02da37d3a291 task_id=8d79611d-38c9-4be2-8a0f-2f3cf5b950f2 user_id=3
2026-10-25T11:07:23.537388Z [info     ] Task pydotorg.celery.run_management_command[8d79611d-38c9-4be2-8a0f-2f3cf5b950f2] succeeded in 0.03519066700027906s: None [celery.app.trace] data={'id': '8d79611d-38c9-4be2-8a0f-2f3cf5b950f2', 'name': 'pydotorg.celery.run_management_command', 'return_value': 'None', 'runtime': 0.03519066700027906, 'args': "['check', [], {}]", 'kwargs': '{}'} request_id=e01e8ff3-9a27-44e9-887e-02da37d3a291 task_id=8d79611d-38c9-4be2-8a0f-2f3cf5b950f2 user_id=3

Failure - before

2026-10-25T11:06:12.251621Z [info     ] Task pydotorg.celery.run_management_command[9512ca93-32ae-4c4c-bb2a-380ac0a87848] received [celery.worker.strategy] data={'id': '9512ca93-32ae-4c4c-bb2a-380ac0a87848', 'name': 'pydotorg.celery.run_management_command', 'args': "['no_such_command', [], {}]", 'kwargs': '{}', 'eta': None}
2026-10-25T11:06:12.256425Z [error    ] Task pydotorg.celery.run_management_command[9512ca93-32ae-4c4c-bb2a-380ac0a87848] raised unexpected: CommandError("Unknown command: 'no_such_command'") [celery.app.trace] data={'hostname': 'celery@local', 'id': '9512ca93-32ae-4c4c-bb2a-380ac0a87848', 'name': 'pydotorg.celery.run_management_command', 'exc': 'CommandError("Unknown command: \'no_such_command\'")', 'traceback': 'Traceback (most recent call last):\n  File "pythondotorg/.venv/lib/python3.14/site-packages/django/core/management/__init__.py", line 110, in call_command\n    app_name = get_commands()[command_name]\n               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^\nKeyError: \'no_such_command\'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "pythondotorg/.venv/lib/python3.14/site-packages/celery/app/trace.py", line 585, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ~~~^^^^^^^^^^^^^^^^^\n  File "pythondotorg/.venv/lib/python3.14/site-packages/celery/app/trace.py", line 858, in __protected_call__\n    return self.run(*args, **kwargs)\n           ~~~~~~~~^^^^^^^^^^^^^^^^^\n  File "pythondotorg/pydotorg/celery.py", line 27, in run_management_command\n    management.call_command(command_name, *args, **kwargs)\n    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "pythondotorg/.venv/lib/python3.14/site-packages/django/core/management/__init__.py", line 112, in call_command\n    raise CommandError("Unknown command: %r" % command_name)\ndjango.core.management.base.CommandError: Unknown command: \'no_such_command\'\n', 'args': "['no_such_command', [], {}]", 'kwargs': '{}', 'description': 'raised unexpected', 'internal': False}
Traceback (most recent call last):
  File "pythondotorg/.venv/lib/python3.14/site-packages/django/core/management/__init__.py", line 110, in call_command
    app_name = get_commands()[command_name]
               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'no_such_command'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pythondotorg/.venv/lib/python3.14/site-packages/celery/app/trace.py", line 585, in trace_task
    R = retval = fun(*args, **kwargs)
                 ~~~^^^^^^^^^^^^^^^^^
  File "pythondotorg/.venv/lib/python3.14/site-packages/celery/app/trace.py", line 858, in __protected_call__
    return self.run(*args, **kwargs)
           ~~~~~~~~^^^^^^^^^^^^^^^^^
  File "pythondotorg/pydotorg/celery.py", line 27, in run_management_command
    management.call_command(command_name, *args, **kwargs)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pythondotorg/.venv/lib/python3.14/site-packages/django/core/management/__init__.py", line 112, in call_command
    raise CommandError("Unknown command: %r" % command_name)
django.core.management.base.CommandError: Unknown command: 'no_such_command'

Failure - after

2026-10-25T11:07:46.757349Z [info     ] Task pydotorg.celery.run_management_command[54e04593-ba2b-4a0b-997a-74dd331dde5e] received [celery.worker.strategy] data={'id': '54e04593-ba2b-4a0b-997a-74dd331dde5e', 'name': 'pydotorg.celery.run_management_command', 'args': "['no_such_command', [], {}]", 'kwargs': '{}', 'eta': None}
2026-10-25T11:07:46.758601Z [info     ] task_started                   [django_structlog.celery.receivers] request_id=f9ff920e-6dc8-49f3-af8b-70c6162e66cd task=pydotorg.celery.run_management_command task_id=54e04593-ba2b-4a0b-997a-74dd331dde5e user_id=3
2026-10-25T11:07:46.764981Z [error    ] task_failed                    [django_structlog.celery.receivers] duration_ms=6 error="Unknown command: 'no_such_command'" request_id=f9ff920e-6dc8-49f3-af8b-70c6162e66cd task_id=54e04593-ba2b-4a0b-997a-74dd331dde5e user_id=3
Traceback (most recent call last):
  File "pythondotorg/.venv/lib/python3.14/site-packages/django/core/management/__init__.py", line 110, in call_command
    app_name = get_commands()[command_name]
               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'no_such_command'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pythondotorg/.venv/lib/python3.14/site-packages/celery/app/trace.py", line 585, in trace_task
    R = retval = fun(*args, **kwargs)
                 ~~~^^^^^^^^^^^^^^^^^
  File "pythondotorg/.venv/lib/python3.14/site-packages/celery/app/trace.py", line 858, in __protected_call__
    return self.run(*args, **kwargs)
           ~~~~~~~~^^^^^^^^^^^^^^^^^
  File "pythondotorg/pydotorg/celery.py", line 29, in run_management_command
    management.call_command(command_name, *args, **kwargs)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pythondotorg/.venv/lib/python3.14/site-packages/django/core/management/__init__.py", line 112, in call_command
    raise CommandError("Unknown command: %r" % command_name)
django.core.management.base.CommandError: Unknown command: 'no_such_command'
2026-10-25T11:07:46.765865Z [error    ] Task pydotorg.celery.run_management_command[54e04593-ba2b-4a0b-997a-74dd331dde5e] raised unexpected: CommandError("Unknown command: 'no_such_command'") [celery.app.trace] data={'hostname': 'celery@local', 'id': '54e04593-ba2b-4a0b-997a-74dd331dde5e', 'name': 'pydotorg.celery.run_management_command', 'exc': 'CommandError("Unknown command: \'no_such_command\'")', 'traceback': 'Traceback (most recent call last):\n  File "pythondotorg/.venv/lib/python3.14/site-packages/django/core/management/__init__.py", line 110, in call_command\n    app_name = get_commands()[command_name]\n               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^\nKeyError: \'no_such_command\'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "pythondotorg/.venv/lib/python3.14/site-packages/celery/app/trace.py", line 585, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ~~~^^^^^^^^^^^^^^^^^\n  File "pythondotorg/.venv/lib/python3.14/site-packages/celery/app/trace.py", line 858, in __protected_call__\n    return self.run(*args, **kwargs)\n           ~~~~~~~~^^^^^^^^^^^^^^^^^\n  File "pythondotorg/pydotorg/celery.py", line 29, in run_management_command\n    management.call_command(command_name, *args, **kwargs)\n    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "pythondotorg/.venv/lib/python3.14/site-packages/django/core/management/__init__.py", line 112, in call_command\n    raise CommandError("Unknown command: %r" % command_name)\ndjango.core.management.base.CommandError: Unknown command: \'no_such_command\'\n', 'args': "['no_such_command', [], {}]", 'kwargs': '{}', 'description': 'raised unexpected', 'internal': False} request_id=f9ff920e-6dc8-49f3-af8b-70c6162e66cd task_id=54e04593-ba2b-4a0b-997a-74dd331dde5e user_id=3
Traceback (most recent call last):
  File "pythondotorg/.venv/lib/python3.14/site-packages/django/core/management/__init__.py", line 110, in call_command
    app_name = get_commands()[command_name]
               ~~~~~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'no_such_command'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "pythondotorg/.venv/lib/python3.14/site-packages/celery/app/trace.py", line 585, in trace_task
    R = retval = fun(*args, **kwargs)
                 ~~~^^^^^^^^^^^^^^^^^
  File "pythondotorg/.venv/lib/python3.14/site-packages/celery/app/trace.py", line 858, in __protected_call__
    return self.run(*args, **kwargs)
           ~~~~~~~~^^^^^^^^^^^^^^^^^
  File "pythondotorg/pydotorg/celery.py", line 29, in run_management_command
    management.call_command(command_name, *args, **kwargs)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pythondotorg/.venv/lib/python3.14/site-packages/django/core/management/__init__.py", line 112, in call_command
    raise CommandError("Unknown command: %r" % command_name)
django.core.management.base.CommandError: Unknown command: 'no_such_command'

PS: Each failure now prints the traceback twice, once under "task_failed" and once under Celery's raised unexpected. Left as is.

Alc-Alc · 2026-10-25T11:28:12Z

 __pycache__
 *.db
 *.py[co]
+*.egg-info/


can remove if it's not wanted, a local uv sync created these, and most Python repos (that I know of) ignore *.egg-info/, so I added it.

jrobichaud · 2026-10-25T14:07:10Z

Disclaimer: I have no pretension of being an expert in structured logging.

However, I believe you might to consider changing your logs to maximize the power of structured logging. This is probably part of a next step of your plan so I understand that you did not change anything at this point.

I just want to make sure you did not miss any important things for your big plan.

Note: Changing logs is a breaking change to whatever log alerts you would have.

Here's a few recommendation:

The log's message should be treated like a unique ID for the log.

I noticed your log's message is Task pydotorg.celery.run_management_command[54e04593-ba2b-4a0b-997a-74dd331dde5e] raised ...

Ex: "task_failed" instead of f"Task {task_name} ...".

It helps to locate the log in the code
It helps to find logs of this given type
It helps to configure alerts accordingly

Use the metadata/kwargs of the structlog logging api to add what you would have included in your format earlier.

Ex: task_name=task.name, task_id=task.id
You can normally craft queries in your logging tool that allows to filter logs based on these values

django-structlog adds a multiple of them.

See:

https://www.structlog.org/en/stable/getting-started.html#your-first-log-entry

Bind values that you want to be repeated on your logs.

Binding values allows to repeat these metadata to each logs allowing to correlate logs together.

django-structlog's request middleware automatically binds the request_id to the request's logs.
django-structlog' celery integration automatically bind the task_id, request_id

django-structlog' celery integration automatically bind the task_id, request_id and I few more things that allows to trace all the things that happened in a single request or in a task.

So if you notice something odd with a given task execution, you can get all the logs of the task using task_id or even to start from the request using request_id.

see:

Customisation

You can customize the behaviour of a few things using django-structlog signals and configurations.

django-structlog's signals documentation: https://django-structlog.readthedocs.io/en/latest/api_documentation.html
an how to list often asked by users: https://django-structlog.readthedocs.io/en/latest/how_tos.html

If there are things you'd like to improve or you face any problems, please open an issue or a PR.

Alc-Alc added 3 commits October 25, 2026 16:32

build(deps): add celery extra to django-structlog

5ac0266

feat(logging): enable django-structlog for celery tasks

5a58adb

chore(gitignore): ignore egg-info build artifacts

5cd8dd4

Alc-Alc requested a review from JacobCoffee as a code owner October 25, 2026 11:26

Alc-Alc commented Oct 25, 2026

View reviewed changes

Navigation Menu

Repository navigation

enable `django-structlog`'s celery integration - #3199

enable `django-structlog`'s celery integration#3199
Alc-Alc wants to merge 3 commits into
python:mainfrom
Alc-Alc:structlog-celery

Alc-Alc commented Oct 25, 2026

Uh oh!

Alc-Alc Oct 25, 2026

Uh oh!

jrobichaud commented Oct 25, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Alc-Alc commented Oct 25, 2026

Description

Closes

Before / After

Uh oh!

Alc-Alc Oct 25, 2026

Choose a reason for hiding this comment

Uh oh!

jrobichaud commented Oct 25, 2026

The log's message should be treated like a unique ID for the log.

Use the metadata/kwargs of the structlog logging api to add what you would have included in your format earlier.

Bind values that you want to be repeated on your logs.

Customisation

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants